HDDS-15342. Make RocksDB bottommost level compaction options configurable for background compaction by ptlrs · Pull Request #10419 · apache/ozone

ptlrs · 2026-06-03T01:09:03Z

What changes were proposed in this pull request?

RocksDB allows multiple configurations for configuration the compaction strategy for the bottom-most layer. These configurations impact the compaction times.

This PR makes the compaction strategy configurable.

Adds a config for setting the compaction strategy
Sets the default strategy to kSkip which is faster than kForce
Adds tests

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15342

How was this patch tested?

CI: https://github.com/ptlrs/ozone/actions/runs/26857346964

…igurability.

sarvekshayr

Thanks @ptlrs for working on this.

…configuration logic.

…nfiguration and update its default type.

sarvekshayr · 2026-06-10T06:29:42Z

Failed acceptance (tools) CI - CompactOMDB defaults to kSkip which does not reduce the OM DB size in this test. #10428 should fix this test by passing the compaction type as kForce and be merged first.

Compact OM DB Column Family
    [Arguments]        ${column_family}
    Execute    ozone repair om compact --cf=${column_family} --service-id ${OM_SERVICE_ID} --node-id om1 --blc 2

ashishkumar50

@ptlrs Thanks for the update, LGTM.

ashishkumar50 · 2026-06-16T08:29:32Z

@ptlrs Can you please look at test failure, looks related.

ptlrs · 2026-06-16T16:54:36Z

@ashishkumar50 as @sarvekshayr had also observed, this PR depends on #10428 which fixes the failing Robot test.

Can we please merge that first? I may need to pull the changes in this PR after that.

…ocksDB-BottommostLevelCompaction-options-configurable # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/CompactDBUtil.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/CompactionService.java

… in OzoneManager.

ptlrs · 2026-06-17T18:42:22Z

Hi @ashishkumar50 @sarvekshayr, could you please take another look? I have merged the changes from master. The robot test should now pass.

adoroszlai · 2026-06-17T20:26:13Z

    </description>
  </property>
+  <property>
+    <name>ozone.om.compaction.service.bottommostlevelcompaction</name>


Please use - as separator in last part of the property name.

Is this necessary? bottommostlevelcompaction is the original name as per org.rocksdb.CompactRangeOptions.BottommostLevelCompaction.

Most importantly, searching for bottommostlevelcompaction brings up all instances of it in both code and configuration in my IDE. Adding a separator would break that.

BottommostLevelCompaction uses CamelCase. The same with lower case needs some separator, e.g. - (kebab-case) or _ (snake_case) for readability. In config property names - is commonly used.

github-actions · 2026-07-09T00:24:08Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

sarvekshayr · 2026-07-09T09:32:40Z

@ptlrs Could you please drive this PR to completion by addressing the minor comments given?

…ocksDB-BottommostLevelCompaction-options-configurable

…BottommostLevelCompaction configuration values.

sarvekshayr

Thanks for updating the patch @ptlrs.

Please look into the CI build failure.
Let's update compaction CLI to also use enum names instead of int - #10428
It can also be done in a follow-up PR.

…on with enum and update configuration handling.

ptlrs

I have updated the CLIs to use the enum directly instead of ints or strings.

${COMPLETION-CANDIDATES} will render the valid values automatically.

bash-5.1$ ozone repair om compact -h                 
Usage: ozone repair om compact [-hV] [--dry-run] [--force] [--verbose]
                               [--blc=<bottommostLevelCompaction>]
                               --cf=<columnFamilyName> [--node-id=<nodeId>]
                               [--service-id=<omServiceId>]
CLI to compact a column family in the om.db. The compaction happens
asynchronously. Requires admin privileges. OM should be running for this tool.
      --blc, --bottommost-level-compaction=<bottommostLevelCompaction>
                           BottommostLevelCompaction option for RocksDB
                             compaction.  Valid values: kSkip,
                             kIfHaveCompactionFilter, kForce, kForceOptimized
                             Default: kSkip
      --cf, --column-family=<columnFamilyName>

bash-5.1$ ozone repair om compact --cf keyTable --blc test1
Invalid value for option '--bottommost-level-compaction': expected one of [kSkip, kIfHaveCompactionFilter, kForce, kForceOptimized] (case-sensitive) but was 'test1'
Usage: ozone repair om compact [-hV] [--dry-run] [--force] [--verbose]

Also, the maven verify command is incorrectly flagging an unused dependency. It only sees the transitive dependency's usage as per the bytecode and ignores the parent dependency's direct usage.

adoroszlai · 2026-07-16T10:16:48Z

+            <!--
+              OMConfigKeys references ManagedCompactRangeOptions.BottommostLevelCompaction, an enum
+              inherited from org.rocksdb.CompactRangeOptions. Bytecode resolves it to the rocksdb
+              class, so the analyzer does not see the hdds-managed-rocksdb reference.
+            -->
+            <ignoredUnusedDeclaredDependency>org.apache.ozone:hdds-managed-rocksdb:jar</ignoredUnusedDeclaredDependency>


Thanks @ptlrs for updating the patch.

Also, the maven verify command is incorrectly flagging an unused dependency. It only sees the transitive dependency's usage as per the bytecode and ignores the parent dependency's direct usage.

A better way to fix it is:

allow usage of CompactRangeOptions.BottommostLevelCompaction, since it's just an enum, safe to use directly:

ozone/pom.xml

Lines 1967 to 1969 in 1a94394



<allowedImport>org.rocksdb.ColumnFamilyDescriptor</allowedImport>

<allowedImport>org.rocksdb.CompactionStyle</allowedImport>

import and use CompactRangeOptions.BottommostLevelCompaction (in OMConfigKeys and elsewhere) instead of ManagedCompactRangeOptions.BottommostLevelCompaction

remove unused dependency hdds-managed-rocksdb from hadoop-ozone/common/pom.xml

Thanks for the suggestion @adoroszlai. I have updated the PR.

…ns and clean up unused dependencies.

adoroszlai · 2026-07-16T18:23:52Z

                    <allowedImports>
                      <!-- Allow non-RocksObject classes. -->
                      <allowedImport>org.rocksdb.ColumnFamilyDescriptor</allowedImport>
+                      <allowedImport>org.rocksdb.CompactRangeOptions</allowedImport>


Sorry for nitpicking: CompactRangeOptions extends RocksObject, so it should not be allowed, only CompactRangeOptions.BottommostLevelCompaction.

ptlrs added 2 commits June 2, 2026 17:35

HDDS-15342. Make RocksDB BottommostLevelCompaction options configurable.

8c35336

HDDS-15342. Add unit tests for RocksDB BottommostLevelCompaction conf…

7281953

…igurability.

ashishkumar50 reviewed Jun 8, 2026

View reviewed changes

Comment thread hadoop-hdds/common/src/main/resources/ozone-default.xml Outdated

sarvekshayr reviewed Jun 8, 2026

View reviewed changes

HDDS-15342. Refactor to centralize RocksDB BottommostLevelCompaction …

059ac20

…configuration logic.

ptlrs mentioned this pull request Jun 9, 2026

HDDS-15466. Make RocksDB bottommost level compaction options configurable for CLI tools #10428

Merged

amaliujia reviewed Jun 9, 2026

View reviewed changes

Comment thread hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/CompactDBUtil.java Outdated

Comment thread hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java

HDDS-15342. Refactor to use getInt for BottommostLevelCompaction co…

422c85c

…nfiguration and update its default type.

ptlrs requested review from amaliujia, ashishkumar50 and sarvekshayr June 9, 2026 18:59

sarvekshayr marked this pull request as ready for review June 10, 2026 05:20

sarvekshayr reviewed Jun 10, 2026

View reviewed changes

Comment thread hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/CompactDBUtil.java

ashishkumar50 reviewed Jun 10, 2026

View reviewed changes

HDDS-15342. Update compaction configuration documentation

d216a87

ashishkumar50 approved these changes Jun 12, 2026

View reviewed changes

ptlrs added 2 commits June 17, 2026 10:23

HDDS-15342. Remove redundant BottommostLevelCompaction initialization…

d74e074

… in OzoneManager.

adoroszlai reviewed Jun 17, 2026

View reviewed changes

github-actions Bot added the stale label Jul 9, 2026

github-actions Bot removed the stale label Jul 10, 2026

Merge remote-tracking branch 'upstream/master' into HDDS-15342-Make-R…

a0d9c5a

…ocksDB-BottommostLevelCompaction-options-configurable

ptlrs requested a review from adoroszlai July 13, 2026 22:56

HDDS-15342. Refactor tests and improve handling of invalid or custom …

02e431e

…BottommostLevelCompaction configuration values.

sarvekshayr reviewed Jul 14, 2026

View reviewed changes

HDDS-15342. Refactor to replace integer-based BottommostLevelCompacti…

ec91d50

…on with enum and update configuration handling.

ptlrs requested a review from sarvekshayr July 16, 2026 04:10

ptlrs commented Jul 16, 2026

View reviewed changes

adoroszlai reviewed Jul 16, 2026

View reviewed changes

HDDS-15342. Replace ManagedCompactRangeOptions with CompactRangeOptio…

2c3c8b4

…ns and clean up unused dependencies.

ptlrs requested a review from adoroszlai July 16, 2026 17:58

adoroszlai reviewed Jul 16, 2026

View reviewed changes

	<!-- Allow non-RocksObject classes. -->
	<allowedImport>org.rocksdb.ColumnFamilyDescriptor</allowedImport>
	<allowedImport>org.rocksdb.CompactionStyle</allowedImport>

Uh oh!

Conversation

ptlrs commented Jun 3, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

sarvekshayr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sarvekshayr commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashishkumar50 left a comment

Choose a reason for hiding this comment

Uh oh!

ashishkumar50 commented Jun 16, 2026

Uh oh!

ptlrs commented Jun 16, 2026

Uh oh!

ptlrs commented Jun 17, 2026

Uh oh!

Uh oh!

adoroszlai Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrs Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

sarvekshayr commented Jul 9, 2026

Uh oh!

sarvekshayr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ptlrs left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrs Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sarvekshayr commented Jun 10, 2026 •

edited

Loading

adoroszlai Jul 14, 2026 •

edited

Loading

sarvekshayr left a comment •

edited

Loading